Action Understanding as Inverse Planning Appendix

This section formalizes the encoding of an agent’s environment and goal into a Markov decision problem (MDP), and describes how this MDP can be solved efficiently by algorithms for rational planning. Let π be an agent’s plan, referred to here (and in the MDP literature) as a policy, such that Pπ(at|st, g, w) is a probability distribution over actions at at time t, given the agent’s state st at time t, the agent’s goal g and world state w. This distribution formalizes P (Actions|Goal, Environment), the expression for probabilistic planning sketched in the main text. The policy π encodes all goal-dependent plans the agent could make in a given environment. We assume that agents’ policies follow the principle of rationality. Within a goal-based MDP, this means that agents choose action sequences that minimize the expected cost to achieve their goals, given their beliefs about the environment. Let Cg,w(a, s) be the environmentand goal-dependent cost to an agent of taking action a in state s. The expected cost to an agent of executing policy π starting from state s is given by the agent’s value function, which sums the costs the agent is expected to incur over an infinite horizon: